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A Contrastive Qualitative Evaluation of Two Different Sequential 
Programs Launched at the School of Foreign Languages of a 

Turkish University 


In a year-long preparatory English class in a state university in Turkey, students are expected to 
develop knowledge and skills related to language including grammar, vocabulary, listening, reading, 
writing ,and speaking. In this context students and teachers are challenged in considering how much 
accuracy should be emphasized in the learning and teaching process, which leads to rethinking and 
reframing to what extent and how grammar teaching/learning should take place in the language 
program. Having been challenged with these questions, this paper presents an evaluation of two 
different regimes of grammar teaching, one that follows its own language content and another that 
follows the language content of the main course. To determine the value of the programs, the 
administrator, 5 instructors and 36 students were interviewed with one-on-one and group interview 
techniques and 6 classroom hours were observed and relevant exam results and attendance records 
were compared. It was found that the program following the language content of the main course is 
favored much more strongly as a positive step in the course of producing the desired learning 
outcomes. 

Keywords: Program evaluation; language teaching program; grammar teaching. 


With especially the emergence of the communicative principles in the 1970s, the recent decades have 
seen various arguments for and against grammar instruction in language teaching. After being 
underestimated and even claimed to be deleterious for a considerable time, it has become clear that 
grammar is a "tool or resource to be used in the comprehension and production of oral and written 
discourse" (Celce-Murica, 1991, p.466) and recent research has demonstrated the need for formal 
instruction for learners to attain high levels of accuracy and proficiency and this has led to a 
resurgence of grammar teaching (Nassaji & Fotos, 2004, p.l26). 
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The reconsideration of grammar teaching in language classroom is rooted in currently favored 
language acquisition studies, which consider language learning without consciousness of form highly 
questionable. Schmidt (1993) suggests that conscious attention to form, i.e. noticing, is a prerequisite 
for language learning. Investigators such as Skehan (1998) have supportingly reported that it is 
necessary for learners to notice target forms in input; otherwise they process input for meaning only 
and do not attend to specific forms, and consequently fail to process and acquire them (Nassaji and 
Fotos, 2004, p. 128). Besides, as related research has substantiated, focusing on forms in some way 
is essential for high accuracy levels (Ellis, 2002; Mitchell, 2000) and explicitly presenting the structure, 
exemplifying it and giving rules for its use culminates in durable gains in the learning of target 
structures in comparison to implicit instruction (Norris & Ortega, 2000). 

As for the manner of grammar teaching, while its exact nature is unclear yet, Ellis (2002) points to the 
current research that support the need for combining and harmonizing form focused instruction and 
meaningful communication opportunities with output, collaborative practice and interactional feedback 
mechanisms secured (Nassaji & Fotos, 2004). 

Preparatory English Classes and the Teaching of Grammar: Case in Turkey 

In Turkey, some state universities and all private universities where medium of instruction is English 
offer preparatory classes. On the other hand, in some other state universities such as Uludag 
University (Bursa), students from some departments where 30 % of the undergraduate compulsory 
courses are delivered in English are obliged to go through a preparatory year (for the regulations see 
http://www.yok.gov.tr/content/view/1047). 

The general purpose of the English preparatory year is to introduce grammar and vocabulary and help 
students to develop skills of aural comprehension, speaking, reading and writing. Usually this objective 
is met by designing separate courses for these language areas and sometimes by merging some of 
these in one course, such as speaking-listening. In this context, grammar is mostly taught as part of 
the content of the main course, which in fact includes all the skill areas but puts more emphasis on 
explicit grammar learning. In the preparatory year students usually receive 24-30 hours of English 
instruction per week depending on the proficiency level they enroll. 

Research studies on English preparatory year in Turkey reveal certain positive and negative aspects 
inherent in such programs. For example, Ozkanal and Flakan's (2010) study investigated the 
effectiveness of Eski§ehir Osmangazi University English Preparatory Program as perceived by the 
students enrolled and the results suggested overall contentment with the program and its instructors, 
with the agreed-upon deficiencies that the physical conditions were not satisfactory and there was no 
component for English for specific purposes. Similarly, Tung (2009) studied the effectiveness of the 
English Language Teaching Program at Atilim University, Preparatory School. The findings pointed to 
the need for targeting all language skills and making the materials, instruction and exams compatible 
with each other. Another sample study that can be cited is Karata§ and Fer's (2009), which evaluated 
the English curriculum at Yildiz Technical University Preparatory School. The major findings indicated 
that it was essential that the audio-visual materials had to be varied and the students' needs related 
with their fields and English knowledge necessary for business life had to be determined. In line with 
the findings of these studies, Tung's (2010) study also investigated effectiveness of Ankara University 
Preparatory School program through the perspectives of instructors and students and the findings 
showed that the program partially served for its purpose but some improvements in the physical 
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conditions, content, materials and assessment dimensions were required to make the program more 
effective. 

As this brief literature indicates, English preparatory year programs face several challenges in general. 
There are certainly many reasons lying behind these problems. For instance, it is a fact that these 
programs serve young adults who have diverse needs and expectations. They come from different 
educational backgrounds and their motivational drives also differ (Geng & Kaya, 2010), thus making it 
impossible to have somewhat homogeneous classes, a circumstance which negatively influences 
student achievement of the learning outcomes. Moreover, due to excessive number of students in 
English preparatory classes, insufficient number of instructors and poor physical conditions as cited as 
major handicaps by aforementioned studies, the quality of the programs usually decreases (Tung, 
2010; Ozkanal & Hakan, 2010). 

One of the central issues of these programs has always been the emphasis put on grammar and the 
relatively much time allocated to grammar teaching/learning. In such cases courses that reinforce the 
productive skills of speaking and writing, which require more student-student and student-teacher 
interaction and intensive/ongoing feedback on student progress, are usually neglected. In such a 
context, grammar learning and teaching inevitably becomes one of the main focuses of the programs. 
Besides, 'a need for more speedy processing of language', as stated by Leki and Carson (1994, p. 90), 
often leads students to place more emphasis on grammar courses. Last but not least, the pressure to 
test large number of students in a relatively short period of time objectively gives way to the 
development and implementation of summative and/or formative tests based largely on techniques 
and items measuring grammatical accuracy at the expense of requiring productive and communicative 
skills. As Mendelsohn (1989) puts it, this impossibility or failure to design a communicative test of 
what is to be taught or being taught and the resulting grammar-based discrete-item tests have a 
backwash effect on the teaching, which is heightened by the central nationwide language proficiency 
exams prepared with similar features for any student who would need to certify how proficient he/she 
is during or after his/her higher education. As expected then, the preparatory foreign language higher 
schools in Turkey predominantly offer explicit grammar courses in varying regimes. 

Within this context, this particular study attempts to determine the outcomes and effects of two 
regimes of grammar teaching that were followed in the English preparatory year program at Uludag 
University. With this aim in mind, the study tries to find answers to the following question: 

In comparison with the grammar program that foiiowed its own content, what are the outcomes of 
the grammar program aiigned with the main course content by a curricuiar change? 


Methodology 

This study was mainly designed as a case study and being as such, it mainly followed a qualitative 
approach to find answers to the research problem posed. Scheerens et al (2003, p.390) emphasize 
that qualitative approaches have the characteristics of "open" research formats, such as "open" 
interview questions, and "free" observation and a strong dependence on the views of persons that are 
part of the "evaluand (evaluation object)". This is confirmed by Lynch (1996, p.l07) indicating that 
the most common methods for gathering and recording data in a naturalistic qualitative program 
evaluation are observation, interviews and document analysis. In this respect; based on its main data 
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sources, which are interviews and observations, it is possible to mention that the qualitative side of 
the present study is predominant. 

Embedded in the naturalistic paradigm in pursuit of understanding a phenomenon within its context 
free of any manipulation (Guba & Lincoln, 1989; Yuksel, 2010), the emic approach of qualitative 
evaluation is believed to enable the evaluator to deepen his/her understanding of the program, thus 
"accounting more thoroughly for the outcomes of the program" (Van, 2008, p.4). Besides, because of 
the observations of actions in their natural context and interviews with participants, naturalistic 
evaluators can have a good chance to adjust their assumptions according to the data (Goetz & 
LeCompte, 1982 cited in Van, 2008). 

However, the study is not completely distant from quantifiable data. In consideration of the argument 
that suggests a mix of evaluation strategies (Lynch, 1996) like in making use of pencil-and-paper tests 
together with interviews, the study also benefits from a comparable set of data offered by exam 
results and attendance records. The aim here is to make the investigation more effective through 
achieving "triangulation", which is based on the premise that each method reveals different aspects of 
empirical reality (Denzin, 1978, p.28) and multiple methods of data collection and analysis provide 
more grist for the research mill (Patton, 1999, p.ll92). 

Setting 

The locale for this evaluative study is Uludag University Foreign Language Preparatory School, whose 
programs giving 1-year elementary, pre-intermediate and intermediate level language training have 
not been formally evaluated so far. As a first attempt to do that, this study aims to determine the 
outcomes of two different regimes of grammar teaching that the program has adopted so far (see 
Table 1 for details). 


Table 1. Comparison of the Two Regimes of Grammar Teaching 


Grammar 

teaching 

Approach 

Curriculum 

Alignment 

Hours per week 

Materials 

1=‘ 

Regime 

Explicit 

Not aligned; used 
the course book's 
content as the 
curriculum 

Elementary: 10 
Pre-Int. : 7 
Intermdiate: 6 

Grammar Way 
Series 

2nd 

Regime 

Explicit 

Aligned with the 
main course 
content 

9 for all levels 

New English File 
-1- 

Supplementary 
pack developed 
by instructors 


The first regime in question was followed until the end of the 2009-2010 academic year and as the 
table reveals, it was based on the use of an independent grammar textbook imposing its own syllabus 
free of the grammar points studied in the main course lessons taught via a different textbook. The 
second one, on the other hand, was implemented in the academic year of 2010-2011 in which direct 
grammar instruction and practice was provided through a grammar pack compiled by the instructors 
to be studied in parallel with the sequence of the grammar topics that the text book used in the main 
course lessons sets forth. From this aspect, the study can be considered an outcomes-based 
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evaluation of change resulting from a small-scale curriculum renewal. As O'Dowd (2002, p. 23) 
emphasizes, instituting changes without determining their effect on students, teaching staff, 
management, future curriculum changes, and the educational entity itself may doom the curriculum 
innovation to failure. 

Participants 

The participants of the study are one administrator, 6 instructors, 18 students trained with the 
previous system and 18 others trained with the new program. The reason for choosing that particular 
administrator was that it was her by whom the change was originated. The six instructors in question 
were all testing office members who had taught students from all levels in the previous system. The 
first 18-person group consisted of the previous year students from three different faculties, for which 
the preparatory school is compulsory. In each of the three 6-person subgroups consisting of 
elementary, pre-intermediate and intermediate level students, there were 2 students from each 
faculty. The logic was the same in distributing the students in the other 18-person group, which was 
comprised of last year's students. 

Another participating group was a current pre-intermediate class whose lessons were observed by the 
evaluator systematically. They contributed to the study not as interviewees but as the focus of the 
observational tasks. 

Data Collection Techniques and Procedure 

To investigate the research question of the study, one-on-one and focus group interviews and 
observation were conducted. Additionally, to further understand the effects of regime change in 
grammar teaching, test results of the students were also documented and analyzed. The step by step 
procedure for data collection is briefly summarized in Table 2 below. 


Table 2. Procedures for Data Collection 


Steps 

Procedures 

Aim 

1 

One-on-one interview with the 
administrator 

To elicit the motives behind the 
curricular change regarding grammar 
teaching 

2 

One-on-one interview with the 
instructors 

To understand their perceptions about 
the curricular change 

3 

Focus group interviews with 
students 

To understand the student perceptions 
about both of the regimes to see if they 
and those of the administrator and 
instructors cross-validate one another 

4 

observation of classes 

to see the way the theoretical 
framework of the curricular change was 
put into practice 

5 

analysis of the exam results and 
attendance records of the 
previous and last year 

to reinterpret and cross-validate the 
qualitative data with an additional 
perspective based on quantified 
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information 


The preliminary stage of data collection was the one-on-one interview with the abovementioned 
administrator. After the elicitation of the motives behind the change from her, the instructors were 
interviewed individually on the same day in a sequence. With the information obtained from the 
administrator in mind, they were asked to convey their any kind of positive and negative opinions on 
the old and new program. 

In the following two weeks, focus group interviews were carried out with the two student groups to 
see where their views coincided with and differed from those of the administrator, instructors and 
other students. The students were made a central figure of the evaluation considering the fact that, 
while curriculums supposedly exist to serve their interests, learners' preferences, if sought at all, are 
marginalized and their voices are mostly silent in curricular interventions (Brooker and MacDonald, 
1999, p. 84). 

Over the ensuing two weeks, a total of six Main Course 1 and Main Course 2 (3 each) lessons of the 
same class were observed. The instructors of the class were not among those interviewed and did not 
know what the evaluator specifically looked at so that they did not concentrate on giving what they 
thought the evaluator particularly expected to find. With the role of a "passive participant" observer 
(Spradley, 1980 cited in Lynch, 1996, p.l21), the aim was to see the way the theoretical framework of 
the curricular change was put into practice, thus to check if the underlying theory is merely espoused 
("declared" to be implemented) or a one in actual use (Argyris & Schdn, 1989). To record as 
thoroughly as possible what was happening in the observed context, descriptive field notes were 
taken. As it is not possible to observe everything (Patton, 1987), the thematic framework set as based 
on the aims and expectations shared by the related administrator and instructors served as the 
"sensitizing concepts" (Patton, 1987, p.82), which eased the observational tasks. 

As the last step, access was gained to the documents of the program on a computerized database. 
The exam results and attendance records of the last two years, belonging to two different regimes of 
grammar teaching, were examined to reinterpret the qualitative data with an additional perspective 
gained through quantified information. This was done to be able to check the conclusions across 
different kinds of data and thus contribute to validity establishment (Lynch 1996, p.l52). Another 
important point here is that, after getting at the exam results, a follow-up meeting was held with the 
members of the testing office and a consensus was reached that the exams had approximately the 
same level of difficulty. 

Data Analysis 

As Lynch (1996, p.l39) indicates, for focusing the evaluation in qualitative data analysis, a useful way 
is to develop a thematic framework that represents the most important evaluation questions. It is 
used to classify and organize data according to key themes, concepts and emergent categories 
(Ritchie et al., 2003, p. 220). Based the ideas and expectations behind the curricular change shared 
by the administrator as its originator, the thematic framework which was set to focus the interview 
questions and analysis is presented below (see Table 3). 

Table 3. Thematic Framework: U. U. School of Foreign Languages 
1. Student and instructor perception of the old program in relation to the 
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curriculum design 

2. Student and instructor perception of the old program in relation to the teaching 
practices 

3. Student and instructor perception of the old program in relation to skill 
development 

4. Student and instructor perception of the old program in relation to the exams 

5. Student and instructor feelings regarding the old program 

6. Effects of the new program on teaching practices 

7. Effects of the new program on skill development 

8. Effects of the new program on the exams and perception of the exams 

9. Student and instructor attitudes towards the new program with its new 
curriculum design 


The interview data had already been recorded by taking simultaneous notes confirmed by the related 
interviewees and as the second step, using the thematic framework as a guide, the entire data set 
was read through several times to code it. This procedure of recursive data reading was done 
throughout the entire study for every data set as based on the analytical framework of analytic 
induction, according to which the data are read several times in pursuit of the relevant emergent 
patterns and themes (Bogdan & Biklen, 1992) that can be coded and/or categorized. The codes were 
put in the margins of the notepapers to reduce and classify the data. The code system used is below 
(see Table 4). 

Table 4. Codes from the U. U. School of Foreign Languages Program Evaluation 

POPC: perception of the old program curriculum design 

POTP: perception of the old program teaching practices 

POSD: perception of the old program in terms of skill development 

POE: perception of the old program in relation to the exams 

FO: feelings regarding the old program 

ENTP: effects of the new program on teaching practices 

ENSD: effects of the new program on skill development 

ENE: effects of the new program on exams 

AN: attitudes towards the new program 

OTFI: other 


While the coding system revealed the most common themes, some of what the interviewed 
participants said in relation to the questions is directly quoted in the text to explicitly clarify the ideas 
and to support the emerging themes. Therefore, to maintain these participants' anonymity the 
instructors were coded in a range from II to 16 and students in a range from SI to S36 while 
reporting their ideas. 

As for the analysis of the observational data, the field notes taken according to the aforementioned 
notion of "sensitizing concepts" were read through several times to be interpreted. Below is the list of 
the observational foci, about which field notes were taken during the observational tasks according to 
the sensitizing concepts shaped by the interviews with the key administrator and instructors (see 
Table 5). 
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Table 5. The List of the Observational Foci of Classroom Observation at U. U. School 

of Foreign Languages Program 

• Whether any omission occurs in the scheduled materials for work on each skill 

• Teacher and student use of the target language 

• The academic readiness of the students for the topics to be studied 

• The teaching pace 

• Student motivation and collective participation 

• Whether any extra materials can be used for reinforcement and consolidation 


The smoothness of the transition between the lessons of the two instructors to have an idea about the 
parallelism between the courses and cooperation between the instructors 

For the analysis of the quantitative data, two midterm and two quiz averages of all the previous and 
last year elementary, pre-intermediate and intermediate classes were calculated to compare the 
success rates with one another. Besides that, the average 8-week attendance performance of all the 
level classes of the previous and last year was calculated to have an idea about the motivational 
aspect of the two programs. 


Results 


Interview with the Administrator 

This interview had the "standardized open-ended interview" format, which has an exact wording and 
order of questions determined in advance (Lynch, 1996, p.l27) and is appropriate especially when 
limited time for the interviewee is a problem (Patton, 1987; Lynch, 1996). As she was the person who 
introduced the change into the system, this format was preferred also to have a clear picture of the 
theoretical motives behind that curricular intervention and focus the subsequent interviews with the 
instructors and students. The questions asked and the answers are below: 

1) What problems were identified about the old system to introduce this change? 

"The success rates in the exams were not satisfactory and the situation was not any 
different in terms of the national and international language exams. We had two 
independent courses called" 

"Main Course" and "Grammar" with their own instructors and textbooks, and they were 
inevitably taught in a disconnected way right from the first day of school. In order to 
expose the students to the whole theoretical coverage of the two courses in a certain 
period, the teaching practices favored input too much in predominantly teacher and 
iecture-based sessions with limited learner language production. When things had to be 
omitted because of lack of time, writing and speaking activities were always the first This 
caused weariness and boredom for both students and instructors. Despite all these, I can 
say that the grammar points taught neither in the main course nor in the grammar 
course could be consolidated decently. " 

2) Does this curricular intervention have a theoretical and/or practical background? 

"Because I am an instructor as well, I had first-hand practical experience myseif about 
the problems. On the other hand, the curricuiums of 7 different universities were 
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examined and discussed with reiated peopie in congresses. I synthesized this background 
information with what I got from a training program on curricuium renewai provided by 
the American Embassy. " 

3) What do you expect from this change? Do you have any specified concrete objectives? 

"We no more want to taik about any "omitted part" in any teaching materiai. For this, we 
converted the "Grammar" course into "Main Course 2" integrated with "Main Course l"to 
achieve paraiieiism and properiy consoiidate what is iearned in any of the courses. We 
now try to offer a iess detaiied grammaticai coverage to create more time for ianguage 
production. The aim is to teach iess but in a more effective way through muiti-skiii 
deveiopment. We are going to try to support this with more production -oriented and 
contextuaiized exam items. We expect higher grade point averages in the exams and 
increased attendance at the courses. " 

4) Do you consider this year a trial period for the implementation of this change? Do you have any 
established criteria to get back to the old system or revise the new one? 

"Of course this is a year of triai. For instance, we have some concerns about the pack 
that repiaced the independent grammar books used in the oid seif-contained grammar 
course. It was not produced by expert materiai deveiopers. We might need to get heip 
about it to revise or change it. Our main target is muiti-skiii deveiopment and we wiii 
keep working for that I can definiteiy say that we need feedback on what we are trying 
to do. " 

5) Do you plan to get feedback from the practitioners and students throughout the process? 

"It is not systematized but I keep getting informai feedback. Comparabie exam resuits 
and attendance records can serve as a concrete ieading indicator. " 

6) Is there any related thing you would like to add? 

"We pian to carry out detaiied needs anaiysis about our prospective students. It is an 
ideai to shape the education given here according their academic and professionai iife. 

This requires facuity-based investigation and cooperation. Besides this, in accordance 
with the change in the system, we pian to integrate speaking and pronunciation 
components into the exams to measure what we are trying to give. " 

Interviews with the Instructors 

These interviews had the "informal conversational interview" format, in which the interviewer 
attempts to engage the interviewee in a natural conversation with questions arising more or less 
spontaneously (Lynch, 1996, p.l26). In conformity with this format, there was not a predetermined 
set of questions (Patton, 2002, p.342) and a premium was put on discovering the interviewees' 
perceptions of the two programs in their own words (Lynch, 1996, p.l27). Initiation of the interview 
by just asking them to compare the programs of the previous and last year produced a comprehensive 
mass of data. In consideration of the thematic framework already set according to the information 
obtained from the key administrator, some related complementary questions were asked when 
needed. The recurring themes across the interviews are as follows: 

* The coverage of the grammar course was considerably extensive and detailed. 
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* The pace of the grammar course had to be so fast that the academic readiness of especially the 
elementary students could not always be taken into consideration. 

* Within the time given, it was a lot more than most of our students could take and it gave us no 
chance to teach according to communicative language teaching principles. 

* As we were in a rush to theoretically cover all the salient features in the content of the two courses, 
we sacrificed so many activities and they were mainly those calling for language production. We often 
omitted even some parts in the workbook and supplementary materials in the teacher's book. 

* What we had to do in the grammar classes was 'talking about English grammar in Turkish! in a 
mechanical way. Although a considerable amount of time was allocated for grammar, there was 
almost no chance to make it functional with appropriate activities. 

* There was no communication between the instructors of the 'Main Course' and 'Grammar'. It was so 
difficult for it to happen anyway as teachers needed to follow their own syllabuses that included the 
instruction of language points in completely different sequences. 

* In time, the students adopted an attitude according to which knowing a language and/or succeeding 
in exams means knowing a lot about the grammar of that language. 

* The tendency to grammar-based lessons that resulted from the old system created mainly grammar- 
oriented exams. 

Two quotes that can characterize the kind of responses above are as follows: 

"We were teaching in a way just the opposite of what we had been taught to do in our 
language teacher education. " [12 (Instructor 2)] 

"For the students, English almost consisted of gap- filling activities of grammar. There were 
many of them skipping main course classes with the thought that attending grammar 
classes would suffice to achieve in big exams. "[13] 

* No need is felt now to omit things. Besides the speaking course, it is now possible to offer all the 
speaking sessions scheduled for the main course classes. 

* We feel relieved about covering the syllabus. There can be time even for activities like games. 
Confused and bored student faces are fewer now. 

* No urge is felt now to teach in Turkish. 

* When a language point is studied, we and students can practice it more from the pack compatible 
with the course book. Last year, it sometimes took a very long time to restudy a point in the grammar 
book used. 
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* Like last year, there are two courses with different names and different instructors but one carries 
on from the point where the other has stopped. This has brought uniformity and inevitable 
cooperation to revise and consolidate the points studied. 

Two quotes that can characterize this kind of responses are as follows: 

"The motto of the current program can be 'Use EngHsh!"\\l\ 

"This system allows production and recycling. "[16] 

* The pack should be revised and enriched. For some specific points, it fails to provide sufficient 
reinforcement. 

* In order for the new program to be more effective and achieve its objectives, the exams should 
change too. In a way, they should measure more of productive skills so that students and instructors 
try harder about writing and speaking sessions. 

* As some advanced grammar points are not taught any more, learning them is left to only self-study 
for students. This might be questioned in terms of such student needs as achieving in exams like 
KPDS and studying at departments teaching in English. 

Interviews with the Students 

Group interviews were conducted with the students. As the interviewing approach, the "interview 
guide" format was adopted. This approach allows the interviewer to specify a range of questions to be 
covered in advance, formulate the wording of the questions as well as the order for asking them as 
the interview progresses, make efficient use of time and phrase the questions in response to the 
natural flow of the interview conversation (Lynch, 1996: 128). Taking these into consideration, it was 
decided that the interview guide format would be ideal to keep pace with the dynamism of the groups 
as much as possible. The questions were formulated according to the thematic framework set as 
based on the previous interviews with the administrator and instructors. 

In accordance with the fact that typically such interviews involve six to eight participants with similar 
background (McKay, 2006: 52), there were six people in each group of elementary, pre-intermediate 
and intermediate level students of the previous and last year. 

Previous Year's Students: 1^ Regime 

Below are the questions asked to the students of the previous year and the themes shaped as based 
on their agreed-upon responses: 

1) You had main course and grammar classes. How complementary do you think they were? Did you 
consider any of them "more important?" 

The courses were distant from being complementary in any way. Intermediate students were aware 
right from the beginning that the two book series used in the two courses were from different 
publishers anyway. Especially elementary students often felt the need to ask their main course 
instructor to teach about the agenda of the grammar course or vice versa. Intermediate students 
favored main course classes more as they thought that they had already known a lot about grammar 
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and needed to benefit as much as possible from the speaking sessions in main course classes. On the 
other hand, elementary students considered grammar classes almost indispensable as they saw in the 
first quiz that grammar knowledge would make up most of exam success. 

2) Do you believe that sufficient reinforcement was given for the topics studied? Were the materials at 
hand used effectively for that? 

For all the groups, every on-paper thing was done in the grammar book. Most of the time, the topics 
in the main course classes were confined to limited practice after being presented in a particular 
context. The time allocated for main course sometimes did not suffice for workbook and 
supplementary worksheets or checking the related answers properly. It can be said that reinforcement 
was left to the extent the students individually tried to benefit from the self-access material unit. 
Intermediate students were a bit more comfortable in this sense. As they had started ahead of the 
other groups and needed to finish two books fewer, they had more time for everything. 

3) Were you given enough chance to practice the instructed points writing and speaking? 

As the instructors mentioned it, speaking activities and sessions always proved to be the first for 
omission. Elementary and pre-intermediate students added the sacrifices made from listening 
exercises as well and reminded that one should listen and understand first to speak after that. All the 
groups made mention of the writing course of two hours a week but in an agreement that they were 
not motivating and fruitful at all. The reason turned out to be that the writing parts in the exams 
mainly required cloze test performance but not production of paragraphs and/or texts. Therefore, in 
writing classes and writing sessions in the other courses, they did not feel the need to make the effort 
to write from scratch. 

4) What was the predominant medium of instruction? 

It was substantially Turkish in the elementary and pre-intermediate grammar classes. The 
intermediate students could cope with the instruction in English, which was given at a slower pace 
than the others were. A striking quote about this from an elementary student is as follows: 

"The only conversation in English was when we exchanged a 'Hello' at the beginning of 
the classes. It was really interesting why it was not a 'Merhaba'"[S3 (Student 3)] 

5) What about the exams? What did they stimulate you to focus on while the program was being run? 

The way the exams led the students was motivating them to focus on detailed grammatical issues and 
memorize as much of the lexicon as possible. As the end-of-year proficiency exam, which was the 
biggest one, was common for all the students, it is the elementary and pre-intermediate students' 
criticism that it could be considerably hard for them while fairly easy for intermediate students. There 
were no planned omissions for the intermediate level but the intermediate students admitted that they 
learned the omitted parts from the students of the other levels so that they did not concentrate on 
them while studying for the proficiency exam as it was common for all. 
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6) Did you ever feel that the instructors were in a hurry to teach things? 

Especially for the elementary and pre-intermediate students, it was felt almost at all times when the 
exams came closer. They also saw that some arbitrary omissions happened mainly from scheduled 
speaking and listening sessions. A related quote worthy of note is as follows from an elementary 
student: 

"I once asked my teacher why we always skipped such parts. She toid me that we would 
not be able to speak anyway. "[S4] 

7) Did you enjoy being a part of the program? Can you identify the times when you feit bored and 
those when you had fun? 

The feelings were predominantly negative about it. Some extremist comments like "it was just a loss 
of time" were made. Besides, this was the item about which significant in-group conflicts occurred 
especially for the elementary level students. It was understood that they favored main course classes 
most but the conflicts in question made the evaluator think that it was closely related to the 
instructors. 

8) Any other things you want to add? 

The related additions were about the lack of parallelism between the courses and the message they 
got from the program, which can be roughly formulated as "English is its grammar." A pre- 
intermediate level medical student's metaphorical utterance about it may sound impressive: 

"OK, grammar is the skeleton of a language but what is a skeleton good for without the 
body covering /f.^"[Sl 1] 

A quote from another pre-intermediate student, which can give an idea about how the old program 
functioned in general, is as follows: 

"If I need English in the future for an academic or professional reason, I definitely think 
that I will need a new process of training again. "[S9] 

Another recurring theme brought forward was the demand for learning professional English and they 
declared that their academic life now does not necessitate the use of English, which makes the 
preparatory school dysfunctional and their language knowledge doomed to be forgotten. 

Last Year's Students: Regime 

The disadvantage for the students of last year was that they had been exposed to the new program 
for nearly 3 months when the interviews were conducted. For this reason, they evaluated a certain 
period of the program while the students of the previous year gave their responses considering the 
whole year. Below are the questions asked to the students of last year and the themes shaped as 
based on their agreed-upon responses: 

1 ) You have Main Course 1 and Main Course 2 classes. How complementary do you think they are? Do 
you consider any of them "more important?" 
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For all the groups, it was understood that the two courses meant one compact course for the students 
though the documentation reports them as two different courses with different names. As this was the 
case, there does not seem to be a perception of superiority between them and a reason for 
considering the classes of either of them more suitable to be skipped. 

2) Do you believe that sufficient reinforcement is given for the topics studied? Are the materials at 
hand used effectively for that? 

For all the groups, sufficient exercises for revision and consolidation are provided. Besides what is 
offered by the workbooks, worksheets and additional communicative activities; a new term as the 
administration of "quick tests" was produced by the pre-intermediate students. The elementary 
students agreed that such extra reinforcement work sometimes even feels too much. 

3) Are you given enough chance to practice the instructed points writing and speaking? 

There is an agreement between all the groups that there has not been even a single part omitted. The 
pre-intermediate students seemed the most content with the work on speaking and writing and the 
constant provision of feedback on their performance. The intermediate students demand reduction in 
writing and reading lessons and even more work on speaking as they feel that it is their pressing 
need. 

4) What is the predominant medium of instruction? 

There is absolute agreement that English dominates as the medium of instruction. It is understood 
that Turkish is employed as a last resort in times of crisis. 

5) What about the exams? What do they stimulate you to focus on while the program is being run? 

The pre-intermediate and intermediate students do not perceive the exams grammar-oriented. 
Especially the intermediate students emphasized that the writing component is fairly demanding. It is 
important that the elementary students still called the exams grammar-focused like the students of 
the previous year. This can be because there is a concern for objectivity in the grammar level exams 
as their productive skills are rather limited yet. 

6) Do you ever feel that the instructors are in a hurry to teach things? 

The responses of all the groups point to a sharp contrast to those of the students of the previous 
year. It is understood that even some kind of tardiness is what is sometimes perceived. Besides the 
time that can be created for productive sessions; the pre-intermediate students noted that their time 
is enough for extensive reading as well, which helps vocabulary development. 

7) Do you enjoy being a part of the program? Can you identify the times when you fee! bored and 
those when you have fun? 

Generally speaking and when compared to the old program, it can be mentioned that the new 
program is closer to provide language learning as a fun experience. The response of an elementary 
level student can be illuminating here: 
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'We do feel that we are learning the language and it feels really good. "[S21] 

8) Any other things you want to add? 

It is interesting that especially the pre-intermediate and intermediate students have such demands as 
native speaker instructors, end-of-term drama shows in English, a much richer library, speaking clubs, 
training on professional English addressing different fields, special focus on linguistic features like 
phrasal verbs etc. This can be deemed interesting as the students of the previous year did not make 
even mention of any of them while concluding the interviews. This seems to suggest that their needs 
were rather "fundamental". 

The Observations 

Within the total of six Main Course 1 and Main Course 2 (3 each) lessons of the same pre-intermediate 
class observed, no omission in the scheduled materials was observed and all the activities targeted at 
the development of different skills including pronunciation and grammatical and lexical knowledge as 
well. 

As for the use of the target language, it was seen that English dominated as the medium of 
instruction and classroom conversations. However, it was observed that Turkish was sometimes 
employed for teaching and asking about some particular grammar points like passives (the teacher 
felt the need to mention that "passive voice" is the equivalent of "edilgen yapi" in Turkish and gave 
sentences for translation). 

As for the academic readiness of the students; it was seen that "passive voice" was presented with 
only its present and past forms in 3 hours, about which the majority of the students seemed ready 
and did not have any considerable difficulty. The syllabus design in the documentation of the old 
program suggests that "passive voice" was studied with all of its dimensions in 5 hours. 

In all of the six hours observed, no sign of hurry to cover the content was detected although there 
was a quiz looming. It was an obvious attempt to give most of the students the chance to talk. 

When it comes to student motivation; it was observed that no obvious lack of motivation or boredom 
arose from a one-dimensional mechanic and monotonous way of teaching, which was recurrently 
reported about the previous year's grammar courses and also some main course hours whose content 
was impoverished with omitted parts. 

As for the use of extra materials for reinforcement and consolidation, it was seen that the grammar 
and conversational activities in the teacher's book were used when needed. Besides, an instructor 
used a matching game provided from the supplementary materials office. 

As for the last focus point, it was observed that the second instructor started exactly from where the 
other one left. Before doing it, he checked the students about what they remembered from the 
previous hour about some crucial points like turning two-object sentences into passive and this 
conveyed the impression that he had seen the other instructor for cooperative teaching. 
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The Exams and Attendance Records 


Table 6. The Exam Score Averages of the Students of the Previous and Last Year 


Elementary 

Quiz 1 

Midterm 1 

Quiz 2 

Midterm 2 

Average 

2009 - 2010 

57.79 

58.96 

60.29 

44.26 

55.32 

2010-2011 

73.98 

72.57 

68.73 

68.61 

70.97 

Pre-Int. 

Quiz 1 

Midterm 1 

Quiz 2 

Midterm 2 

Average 

2009 - 2010 

73.67 

67.59 

76.7 

67.48 

71.36 

2010-2011 

87.85 

85.71 

70.34 

74.53 

79.60 

Intermediate 

Quiz 1 

Midterm 1 

Quiz 2 

Midterm 2 

Average 

2009 - 2010 

77.7 

66.53 

58.68 

68.80 

67.92 

2010-2011 

78.84 

90.89 

77.79 

74.17 

77.92 

Table 7. The Average Flour of Absenteeism in 8 Weeks of the Students of the Previous 



and Last Year 




Elementary Pre-Intermediate Intermediate 

2009 - 2010 


57.47 

37.66 


32.11 

2010 - 2011 


44.52 

31.72 


29.17 


It is seen that the new program seems to produce higher achievement in exams and result in higher 
and more regular attendance. 

The Measures Taken for Validity 

As Maxwell (1992) states, validity in a naturalistic design is seen as the correspondence between the 
researcher's "account" of some phenomena and their "reality". In order to achieve this; triangulation, 
which refers to the gathering and reconciling of data from several sources and/or from different data- 
gathering techniques, deserves special consideration for validity check (Lynch, 1996, p.59). For 
Greene and McClintock (1985); if the student, the teacher and the administrator in a particular context 
all report the same perception of some aspect of the program, then these viewpoints converge on a 
conclusion whose validity is strengthened. For Lynch (1996, p.62), a main idea in a naturalistic 
evaluation is to arrive at evaluation conclusions as a result of a consensus among persons from 
different perspectives in relation to the program. In consideration of these; the present study 
employed different data collection techniques and elicited the perspectives of different stakeholders 
and according to the results presented above, they seem to suggest converging viewpoints regarding 
both of the programs evaluated. 

As for the other techniques to confirm naturalistic validity, Guba and Lincoln (1989, p. 233-43 cited in 
Lynch, 1996, p.56) provide: 

* prolonged engagement, which is the immersion of the evaluator in the setting to establish "rapport 
and trust" with the participants. It can be mentioned that this was achieved to a considerable extent 
thanks to the personal relationships between the evaluator and the instructors for a couple of years. 

* persistent observation, which is the evaluator's attempt to identify the most relevant elements of the 
evaluation setting. The evaluator tried to do this by starting the evaluation interviewing the 
administrator who originated the curricular intervention in question. The aim here was to get at the 


97 



Turkish Online Journal of Qualitative Inquiry, July 2012, 3(3) 


logic behind the change, what is aimed with it and what is expected from it. This served as a 
foundation and the following evaluative activities were based on the relevant thematic framework. 

* peer debriefing, which is a discussion between the evaluator and a "disinterested peer" concerning 
the evaluation results. This was done with an instructor who has been teaching in the institution for 
10 years and no conflict occurred. 

* member checks, which refers to (formal or informal) repeated checking of evaluation findings with 
the members of the setting who provided the data. This was done in an informal follow-up meeting 
with the instructors and one representative from every participating student group. 

* negative case analysis, which refers to the search by the evaluator for rival explanations about the 
evaluation data that do not fit the "working hypotheses". This was achieved over the views of a 
different administrator in the program, who had not been a participant in the evaluation process. She 
postulated that the positive outcomes that the new program seems to be producing should mainly be 
attributed to the decreased populations of the classes, which means approximately 7 students fewer 
per class. Thereupon several instructors were asked if they would prefer the old program if the classes 
were again made as crowded as they had been in the previous year. All of them kept defending the 
new program reiterating its positive outcomes. What is more, one instructor reminded that a 
considerable number of the previous year students either abandoned or largely neglected the program 
as they eventually ended up hopeless to achieve or planned to pass at summer school. This resulted 
in many classes with a population similar to the ones last year but did not create significantly higher 
achievement. This cyclical approach in data collection and analysis occurred throughout the study, in 
which every stage of collection and analysis was influenced by the data already collected (Borg, 
2005). 


Discussion 

The key evaluation findings of the present study suggest that the implementation of the new program 
is a positive step forward for the school especially when the findings about the previous year's 
program are taken into consideration. 

In the light of the experiences, aims and expectations of the administrator as the originator of the 
change, instructors and students; last year's program seems to be better functioning in the course of 
having parallelism between the courses, teaching at an appropriate pace with effective use of the 
materials, achieving multi-skill development as found by Arkin (2010) to be a positive language 
teaching program aspect, creating a positive learner attitude towards English, generating higher 
motivation, having greater exam success and increasing student attendance, which in fact can all be 
considered interrelated and seem to be the points that the previous program was lacking in. Bearing 
this in mind, it can be asserted that the new program should continue to be run working on its 
components found to be in need of improvement and/or modification and those which need to be 
added in it. 

The investigation of the program documentation showed that there is one more trial of this kind in the 
history of the program. It was learned that it failed because of bad material choice, which makes last 
year's supplementary pack prepared by the program instructors a critical component as it was found 
that it needs to be improved and enriched to more effectively reinforce the acquisition of what is 
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taught through the main textbook. What this would call for can well be theoretical and empirical work 
on material development and evaluation, which can take the form of classroom/action research. 

In the light of another important finding and parallel to those in Tung (2009) and Tung (2010) in 
terms of the need for the compatibility of tests and materials and instruction, it can be postulated that 
the logic behind exam preparation and the exams as another critical program component should 
support the new teaching/learning environment so that both teachers and students feel more 
motivated to work on language production via speaking and writing. It would not be hard to 
understand if some or even most of the learners and instructors in a program reluctantly worked 
towards the production of guided or free written/oral discourse samples as long as they know that 
their performance is going to be assessed with procedures like cloze tests in the end. To eliminate 
this, theoretical and empirical work on testing productive skills can be advisable. 

As another finding worth consideration and echoing those in Ozkanal and Hakan (2010) and Karata§ 
and Fer (2009), the demand for learning professional English and the fact that the previous year's 
students' current academic life does not necessitate the use of English matches up with, as cited in 
Ozkanal and Flakan (2010: p.304), Kinsiz's (2005) emphasis that "foreign language education should 
be interdisciplinary, participatory and innovative" and "the need of professions for foreign language 
knowledge should be met". This idealistic call for English for specific purposes and integrating it into 
the program as a new component should be requiring a macro-level policy and planning based on 
cooperation between the faculties and the preparatory school, duly developing the specific programs 
and pedagogical materials and giving to-the-purpose training to the instructors. 


Conclusion 

This study was the first of its kind in the history of the school and it provided a comparative evaluation 
of two different programs characterized by two different grammar teaching regimes launched in the 
same institution. The last year's program was found to be strongly disfavored by the participants. This 
gets even more important in consideration of the fact that it had been implemented for nearly 8 years. 
If such an evaluative study as this one had been conducted earlier, things could have changed earlier 
too and it is hoped that the present study will serve as a source if it is ever planned to get back to the 
old system or bring about another curricular change in the program evaluated in the first place and 
secondarily in similar programs elsewhere to the extent that they make sense out of the evaluation 
findings. 

The major limitation of the study is that the new program was evaluated as based on its first three 
months while the old one was assessed considering a one-year period. For this reason, a follow-up 
evaluative study at the end of the last year seems to be a need. Another limitation is that there was 
no chance to compare the students as based on the results of some national and international 
language exams. Considering the fact that a peripheral aim of the new program is to increase the 
success level in exams like KPDS, it can be mentioned that having related data would serve much. 
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